Feedforward neural networks from a statistical-modelling perspective

class: title-slide, left, bottom

# Feedforward neural networks from a statistical-modelling perspective
----
## **Andrew McInerney**, **Kevin Burke**
### University of Limerick
#### IWSM, 19 July 2023

---
# Feedforward Neural Networks

--

.left-column[ 
<img src="data:image/png;base64,#img/FNN.png" width="100%" height="110%" style="display: block; margin: auto;" />
]

<br>

--

.right-column[ 
<img src="data:image/png;base64,#img/nneq1.png" width="95%" height="100%" style="display: block; margin: auto 0 auto auto;" />
]

---
count: false
# Feedforward Neural Networks

.left-column[ 
<img src="data:image/png;base64,#img/FNN.png" width="100%" height="110%" style="display: block; margin: auto;" />
]

<br>

.right-column[ 
<img src="data:image/png;base64,#img/nneq2.png" width="95%" height="100%" style="display: block; margin: auto 0 auto auto;" />
]

---
count: false
# Feedforward Neural Networks

.left-column[ 
<img src="data:image/png;base64,#img/FNN.png" width="100%" height="110%" style="display: block; margin: auto;" />
]

<br>

.right-column[ 
<img src="data:image/png;base64,#img/nneq3.png" width="95%" height="100%" style="display: block; margin: auto 0 auto auto;" />
]

---
count: false
# Feedforward Neural Networks

.left-column[ 
<img src="data:image/png;base64,#img/FNN.png" width="100%" height="110%" style="display: block; margin: auto;" />
]

<br>

.right-column[ 
<img src="data:image/png;base64,#img/nneq4.png" width="95%" height="100%" style="display: block; margin: auto 0 auto auto;" />
]

---
count: false
# Feedforward Neural Networks

.left-column[ 
<img src="data:image/png;base64,#img/FNN.png" width="100%" height="110%" style="display: block; margin: auto;" />
]

<br>

.right-column[ 
<img src="data:image/png;base64,#img/nneq5.png" width="95%" height="100%" style="display: block; margin: auto 0 auto auto;" />
]

---

# Data Application

--

### Insurance Data (Kaggle)

--

1,338 beneficiaries enrolled in an insurance plan

--

Response:  `charges`

--

6 Explanatory Variables:
.pull-left[
- `age`

- `sex`

- `bmi`

]

.pull-left[

- `children`

- `smoker`

- `region`

]

---

# R Implementation

--

Many packages available to fit neural networks in R.

<br>
--

Some popular packages are:

--

-  `nnet`

--

-  `neuralnet`

--

-  `keras`

--

-  `torch`

---

# R Implementation: nnet

--

```r
library(nnet)
nn <- nnet(charges ~ ., data = insurance, size = 2, maxit = 2000,
           linout = TRUE)
summary(nn)
```

--

```{.bg-primary}
## a 8-2-1 network with 21 weights
##  b->h1 i1->h1 i2->h1 i3->h1 i4->h1 i5->h1 i6->h1 i7->h1 i8->h1 
##   1.39  -0.43   0.08   0.03  -0.08  -3.16   0.07   0.11   0.15 
##  b->h2 i1->h2 i2->h2 i3->h2 i4->h2 i5->h2 i6->h2 i7->h2 i8->h2 
##   6.31   0.04   0.13   2.19  -0.11  -6.19   0.15   0.12   0.14 
##  b->o h1->o h2->o 
##  1.08 -4.82  2.45 
## [...]
```

---

# Proposed Solution: interpretnn

--

.left-column[ 
<br>
<img src="data:image/png;base64,#img/interpretnn.png" width="80%" style="display: block; margin: auto;" />
]

--

.right-column[
<br>
<br>

```r
# install.packages("devtools")
library(devtools)
install_github("andrew-mcinerney/interpretnn")
```
]

---

# Statistical Perspective

--

$$
y_i = \text{NN}(x_i) + \varepsilon_i, 
$$

--

where

$$
\varepsilon_i \sim N(0, \sigma^2)
$$

<br>

--

$$
\ell(\theta, \sigma^2)= -\frac{n}{2}\log(2\pi\sigma^2)-\frac{1}{2\sigma^2}\sum_{i=1}^n(y_i-\text{NN}(x_i))^2
$$

---

# Uncertainty Quantification

Then, as `\(n \to \infty\)`

$$
\hat{\theta} \sim N[\theta, \Sigma = \mathcal{I}(\theta)^{-1}]
$$

--

Estimate `\(\Sigma\)` using

$$
\hat{\Sigma} = I_o(\hat{\theta})^{-1}
$$

--

<br>

However, inverting `\(I_o(\hat{\theta})\)` can be problematic in neural networks.
---

# Redundancy

--

Redundant hidden nodes can lead to issues of unidentifiability for some of the parameters (Fukumizu 1996).

<br>

--

Redundant hidden nodes `\(\implies\)` Singular information matrix.

<br>

--

Trade-off between model flexibility and interpretability.

---

# Significance Testing

--

.pull-left[
<img src="data:image/png;base64,#img/FNN1.png" width="100%" style="display: block; margin: auto;" />
]

---
count: false
# Significance Testing

.pull-left[
<img src="data:image/png;base64,#img/FNN2.png" width="100%" style="display: block; margin: auto;" />
]

--

.pull-right[
  Wald test:
  
  {{content}}
  
  ]

--

$$
`\begin{equation}
 \omega_j = (\omega_{j1},\omega_{j2},\dotsc,\omega_{jq})^T
\end{equation}`
$$
{{content}}

--

$$
`\begin{equation}
 H_0: \omega_j = 0
\end{equation}`
$$
{{content}}

--

$$
`\begin{equation}
 (\hat{\omega}_{j} - \omega_j)^T\Sigma_{\hat{\omega}_{j}}^{-1}(\hat{\omega}_{j} - \omega_j) \sim \chi^2_q
\end{equation}`
$$
{{content}}

---

# Insurance: Model Summary

```r
intnn <- interpretnn(nn)  
summary(intnn)
```

--

```{.bg-primary}
## Coefficients:
##                                Weights |      X^2    Pr(> X^2)
##               age     (-0.43***, 0.04) |  41.4363 1.01e-09 ***
##          sex.male        (0.08*, 0.13) |   5.5055 6.38e-02 .  
##               bmi      (0.03, 2.19***) | 105.6106 0.00e+00 ***
##          children   (-0.08***, -0.11.) |  19.0146 7.43e-05 ***
##        smoker.yes (-3.16***, -6.19***) | 250.6393 0.00e+00 ***
##  region.northwest        (0.07., 0.15) |   2.8437 2.41e-01    
##  region.southeast        (0.11*, 0.12) |   6.2560 4.38e-02 *  
##  region.southwest       (0.15**, 0.14) |  10.8218 4.47e-03 ** 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
```

---

# Insurance: Model Summary

```r
plotnn(intnn)
```

--

<img src="data:image/png;base64,#img/plotnn-1.png" width="90%" style="display: block; margin: auto;" />

---

# Covariate-Effect Plots

$$
`\begin{equation}
 \widehat{\overline{\text{NN}}}_j(x) = \frac{1}{n}\sum_{i=1}^n \text{NN}(x_{i,1}, \ldots,x_{i,j-1},x, x_{i,j+1}, \ldots)
\end{equation}`
$$

--
Propose covariate-effect plots of the following form:

--

$$
`\begin{equation}
 \hat{\beta}_j(x,d) =\widehat{\overline{\text{NN}}}_j(x + d) -\widehat{\overline{\text{NN}}}_j(x)
\end{equation}`
$$

--

Usually set `\(d = \text{SD}(x_j)\)`

---

# Insurance: Covariate Effects

```r
plot(intnn, conf_int = TRUE, which = c(1, 4))
```

--

.pull-left[ 
![](data:image/png;base64,#fnn-stat_files/figure-html/plot1-1.png)
]

--

.pull-right[ 
![](data:image/png;base64,#fnn-stat_files/figure-html/plot2-1.png)
]

---
# Summary

--

* Statistical-modelling approach to neural networks

<br>
--

* Provide an R package to improve interpretability

<br>
--

* Make neural networks more familiar in a statistical-modelling context

---
class: bigger
# References

*  McInerney, A., & Burke, K. (2022). A statistically-based approach to feedforward neural network model selection. arXiv preprint arXiv:2207.04248.

*  McInerney, A., & Burke, K. (2023). Interpreting feedforward neural networks as statistical models. In Preparation.

### R Package

```r
devtools::install_github("andrew-mcinerney/interpretnn")
```

<svg aria-hidden="true" role="img" viewBox="0 0 496 512" style="height:1em;width:0.97em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:#007DBA;overflow:visible;position:relative;"><path d="M165.9 397.4c0 2-2.3 3.6-5.2 3.6-3.3.3-5.6-1.3-5.6-3.6 0-2 2.3-3.6 5.2-3.6 3-.3 5.6 1.3 5.6 3.6zm-31.1-4.5c-.7 2 1.3 4.3 4.3 4.9 2.6 1 5.6 0 6.2-2s-1.3-4.3-4.3-5.2c-2.6-.7-5.5.3-6.2 2.3zm44.2-1.7c-2.9.7-4.9 2.6-4.6 4.9.3 2 2.9 3.3 5.9 2.6 2.9-.7 4.9-2.6 4.6-4.6-.3-1.9-3-3.2-5.9-2.9zM244.8 8C106.1 8 0 113.3 0 252c0 110.9 69.8 205.8 169.5 239.2 12.8 2.3 17.3-5.6 17.3-12.1 0-6.2-.3-40.4-.3-61.4 0 0-70 15-84.7-29.8 0 0-11.4-29.1-27.8-36.6 0 0-22.9-15.7 1.6-15.4 0 0 24.9 2 38.6 25.8 21.9 38.6 58.6 27.5 72.9 20.9 2.3-16 8.8-27.1 16-33.7-55.9-6.2-112.3-14.3-112.3-110.5 0-27.5 7.6-41.3 23.6-58.9-2.6-6.5-11.1-33.3 2.6-67.9 20.9-6.5 69 27 69 27 20-5.6 41.5-8.5 62.8-8.5s42.8 2.9 62.8 8.5c0 0 48.1-33.6 69-27 13.7 34.7 5.2 61.4 2.6 67.9 16 17.7 25.8 31.5 25.8 58.9 0 96.5-58.9 104.2-114.8 110.5 9.2 7.9 17 22.9 17 46.4 0 33.7-.3 75.4-.3 83.6 0 6.5 4.6 14.4 17.3 12.1C428.2 457.8 496 362.9 496 252 496 113.3 383.5 8 244.8 8zM97.2 352.9c-1.3 1-1 3.3.7 5.2 1.6 1.6 3.9 2.3 5.2 1 1.3-1 1-3.3-.7-5.2-1.6-1.6-3.9-2.3-5.2-1zm-10.8-8.1c-.7 1.3.3 2.9 2.3 3.9 1.6 1 3.6.7 4.3-.7.7-1.3-.3-2.9-2.3-3.9-2-.6-3.6-.3-4.3.7zm32.4 35.6c-1.6 1.3-1 4.3 1.3 6.2 2.3 2.3 5.2 2.6 6.5 1 1.3-1.3.7-4.3-1.3-6.2-2.2-2.3-5.2-2.6-6.5-1zm-11.4-14.7c-1.6 1-1.6 3.6 0 5.9 1.6 2.3 4.3 3.3 5.6 2.3 1.6-1.3 1.6-3.9 0-6.2-1.4-2.3-4-3.3-5.6-2z"/></svg> <font size="5.5">andrew-mcinerney</font>   <svg aria-hidden="true" role="img" viewBox="0 0 512 512" style="height:1em;width:1em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:#007DBA;overflow:visible;position:relative;"><path d="M459.37 151.716c.325 4.548.325 9.097.325 13.645 0 138.72-105.583 298.558-298.558 298.558-59.452 0-114.68-17.219-161.137-47.106 8.447.974 16.568 1.299 25.34 1.299 49.055 0 94.213-16.568 130.274-44.832-46.132-.975-84.792-31.188-98.112-72.772 6.498.974 12.995 1.624 19.818 1.624 9.421 0 18.843-1.3 27.614-3.573-48.081-9.747-84.143-51.98-84.143-102.985v-1.299c13.969 7.797 30.214 12.67 47.431 13.319-28.264-18.843-46.781-51.005-46.781-87.391 0-19.492 5.197-37.36 14.294-52.954 51.655 63.675 129.3 105.258 216.365 109.807-1.624-7.797-2.599-15.918-2.599-24.04 0-57.828 46.782-104.934 104.934-104.934 30.213 0 57.502 12.67 76.67 33.137 23.715-4.548 46.456-13.32 66.599-25.34-7.798 24.366-24.366 44.833-46.132 57.827 21.117-2.273 41.584-8.122 60.426-16.243-14.292 20.791-32.161 39.308-52.628 54.253z"/></svg> <font size="5.5">@amcinerney_</font> <svg aria-hidden="true" role="img" viewBox="0 0 512 512" style="height:1em;width:1em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:#007DBA;overflow:visible;position:relative;"><path d="M0 128C0 92.65 28.65 64 64 64H448C483.3 64 512 92.65 512 128V384C512 419.3 483.3 448 448 448H64C28.65 448 0 419.3 0 384V128zM48 128V150.1L220.5 291.7C241.1 308.7 270.9 308.7 291.5 291.7L464 150.1V127.1C464 119.2 456.8 111.1 448 111.1H64C55.16 111.1 48 119.2 48 127.1L48 128zM48 212.2V384C48 392.8 55.16 400 64 400H448C456.8 400 464 392.8 464 384V212.2L322 328.8C283.6 360.3 228.4 360.3 189.1 328.8L48 212.2z"/></svg> <font size="5.5">andrew.mcinerney@ul.ie</font>